System and method for assessing arousal level of driver of vehicle that can select manual driving mode or automated driving mode

ABSTRACT

An apparatus is provided for assessing arousal level of a driver of a vehicle. The apparatus includes a processor and a memory storing a computer program, which when executed by the processor, causes the processor to perform operations. The operations include (A) acquiring driving mode information that indicates when a driving mode of the vehicle is in a first mode, in which travel control of the vehicle is performed by the driver, or in a second mode, in which at least a part of the travel control of the vehicle is automatically performed. The operations also include (B) determining whether the driving mode of the vehicle is in the first mode or the second mode, (C) executing first steps when the driving mode is determined as being the first mode, and (D) executing second steps when the driving mode is determined as being the second mode.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of pending U.S. application Ser. No.15/847,994 filed on Dec. 20, 2017 and claims priority of Japanese PatentApplication No. 2017-021599 filed Feb. 8, 2017. The disclosure of thesedocuments, including the specifications, drawings and claims areincorporated herein by reference in their entirety.

BACKGROUND 1. Technical Field

The present disclosure relates to an arousal level assessment system andan arousal level assessment method. The present disclosure relatesparticularly to an arousal level assessment system and an arousal levelassessment method for assessing arousal level of a driver of a vehicle.

2. Description of the Related Art

In recent years, automated driving systems that automatically performtravel control of a vehicle such as acceleration, steering, and brakingof the vehicle have been actively researched and developed.

For example, Japanese Unexamined Patent Application Publication No.6-171391 discloses an arousal level estimation device that estimates thearousal level of a driver based on a driving operation which is detectedby a driving operation detection sensor which monitors the drivingoperation by the driver in automated driving by using adaptive cruisecontrol (ACC). Note that the arousal level indicates the scale thatindicates the degree of awakening. For example, in a case wheresleepiness occurs to the driver, the arousal level lowers.

The arousal level estimation device disclosed in Japanese UnexaminedPatent Application Publication No. 6-171391 estimates the arousal levelof the driver by using a fluctuation pattern in the steering angle thatcorresponds to the steering operation by the driver, assumes that thedriver experiences sleepiness in a case where the arousal level of thedriver is estimated to be low, actuates an alarm or the like, andthereby awakes the driver. Further, in a case where the frequency ofalarm actuation becomes a prescribed frequency or more, the vehicle isdecelerated by actuating a brake device of the vehicle or is furtherforcibly stopped. In such a manner, the arousal level estimation devicedisclosed in Japanese Unexamined Patent Application Publication No.6-171391 may realize certain safe travel.

SUMMARY

In one general aspect, the techniques disclosed here feature a systemfor assessing arousal level of a driver of a vehicle, the systemincluding at least one first sensor, at least one second sensor, and aprocessor. The processor (A) acquires driving mode information thatindicates a driving mode of the vehicle, the driving mode beingselectable by the vehicle and/or the driver from a first mode in whichtravel control of the vehicle is performed by the driver and a secondmode in which at least a portion of the travel control of the vehicle isautomatically performed, (B) assesses which of the first mode or thesecond mode is selected based on the driving mode information, (C) in acase where the first mode is selected, (c1) acquires driving informationthat indicates a driving operation by the driver and/or indicates adriving state of the vehicle via the at least one first sensor, (c2)acquires physiological information of the driver via the at least onesecond sensor, and (c3) assesses the arousal level based on the drivinginformation and the physiological information, and (D) in a case wherethe second mode is selected, (d1) acquires the physiological informationvia the at least one second sensor, and (d2) assesses the arousal levelbased on the physiological information without referring to the drivinginformation.

It should be noted that general or specific embodiments may beimplemented as a system, a method, an integrated circuit, a computerprogram, a storage medium, or any selective combination thereof.

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram that illustrates one example of aconfiguration of a driving support device in an embodiment;

FIG. 2A is a block diagram that illustrates one example of a specificconfiguration of a driving operation unit illustrated in FIG. 1;

FIG. 2B is a block diagram that illustrates one example of a specificconfiguration of a surrounding environment recognition unit illustratedin FIG. 1;

FIG. 2C is a block diagram that illustrates one example of a specificconfiguration of a subject vehicle position detection unit illustratedin FIG. 1;

FIG. 3 is a block diagram that illustrates examples of specificconfigurations of a vehicle behavior detection unit and a first arousallevel recognition unit, which are illustrated in FIG. 1;

FIG. 4 is a block diagram that illustrates examples of specificconfigurations of a physiological information detection unit and asecond arousal level recognition unit, which are illustrated in FIG. 1;

FIG. 5 is a flowchart that illustrates an outline of a process that isperformed by an arousal level estimation device in the embodiment;

FIGS. 6A and 6B are flowcharts that illustrate an action that isperformed by the arousal level estimation device in the embodiment; and

FIG. 7 is a block diagram that illustrates one example of aconfiguration of the driving support device in a modification example.

DETAILED DESCRIPTION (Underlying Knowledge Forming Basis of PresentDisclosure)

The underlying knowledge forming basis of the present disclosure willhereinafter be described.

As described above, in recent years, automated driving systems thatautomatically perform travel control of a vehicle such as acceleration,steering, and braking of the vehicle have been actively researched anddeveloped. The automated driving system has automation levels, andwidely-used automation levels are defined in view of division of rolesrelated to driving between a driver and a system. For example, theautomation levels that are defined by the National Highway TrafficSafety Administration of the U.S. Department of Transportation arecategorized into five levels. Manual driving is categorized as level 0,and fully automated driving is categorized as level 4.

At the present time, the automation level of the automated drivingsystems that are installed in many vehicles is level 1. Level 1 isdefined not as automated driving but as safe driving support. Level 1indicates a state where adaptive cruise control (ACC), lane keepingassist (LKA), or the like is independently used for the vehicle.Accordingly, at level 1, the automated driving system performs vehiclecontrol in either one of the front-rear direction or the left-rightdirection of the vehicle, and the driver performs the other vehiclecontrol and monitoring of the traffic situation. Further, at level 1,the responsibility for safe driving is on the driver.

For example, the automated driving system of a vehicle that is equippedwith the ACC automatically performs travel control that causes thevehicle to travel at a preset vehicle speed in a case where a precedingvehicle is not present and that adjusts the vehicle speed so as tomaintain a preset vehicular gap in a case where a preceding vehicle isdetected in the front. While the travel control is automaticallyperformed by the automated driving system, because the driver does nothave to perform pedal operations, the driving operations by the driverare only steering operations and are quite monotonous. Thus, the drivermay become less mindful, and it is possible that the driver performsother work than driving or the arousal level lowers (sleepiness occurs).That is, it is possible that, although the driver is responsible forsafe driving with the automated driving system of the vehicle equippedwith the ACC, the driver excessively trusts the ACC and becomes carelessabout attention to and monitoring of a surrounding environment.

In such a manner, in a case where the automated driving system suddenlyfalls into a functional limit when the consciousness of the driver aboutdriving lowers, it may be considered that the possibility that theresponse of the driver to a target object to which attention has to bepaid in driving is delayed or the driver misses the target objectbecomes high. Further, in the worst case, this possibly results in atraffic accident.

Accordingly, in order to simultaneously satisfy comfort by use of theACC and safe driving by the driver, a technique has been suggested whichmonitors the state of the driver, particularly, the arousal level andalerts the driver in a case where the arousal level becomes lower thanan acceptable value when a function of the automated driving such as theACC is used (For example, Japanese Unexamined Patent ApplicationPublication No. 6-171391).

As described above, Japanese Unexamined Patent Application PublicationNo. 6-171391 discloses an arousal level estimation device that estimatesthe arousal level of a driver based on a driving operation which isdetected by a driving operation detection sensor in automated driving byusing the ACC. Specifically, the arousal level estimation devicedisclosed in Japanese Unexamined Patent Application Publication No.6-171391 monitors a fluctuation pattern in the steering angle of thesteering by the driver and estimates the arousal level of the driverbased on a frequency feature of the pattern. In a case where the arousallevel of the driver is estimated to be low, an alarm or the like isactuated, and the driver is thereby awoken. Further, in a case where thefrequency of alarm actuation becomes a prescribed frequency or more, thevehicle is decelerated by actuating a brake device or is furtherforcibly stopped. In such a manner, the arousal level estimation devicedisclosed in Japanese Unexamined Patent Application Publication No.6-171391 may realize certain safe travel.

At the present time, as a next generation automated driving system,automated driving systems whose automation level is level 2 are beingpartially put to practical use. Because the automated driving system atlevel 2 performs the vehicle control in the front-rear direction and theleft-right direction of the vehicle by fully using of the ACC, the LKA,and so forth, the driver does not have to perform pedal operations orsteering operations. However, because the responsibility for safedriving at level 2 is on the driver similarly to level 1, the driver hasto regularly monitor the automated driving system and the surroundingenvironment in case of the functional limit of the automated drivingsystem.

However, at level 2, because the driver has to perform few drivingoperations as long as the automated driving system stably acts, thepossibility that the consciousness about driving or arousal level lowersbecomes higher than level 1. For example, in a case where the automateddriving system falls into the functional limit or fails, the driver hasto take over the driving operation from the automated driving system.However, in a case where switching is made to manual driving when thearousal level of the driver lowers, it is possible that the response ofthe driver to a target object to which attention has to be paid indriving of the vehicle is delayed or the driver misses the targetobject, and this further results in a traffic accident in the worstcase.

Accordingly, in order to realize the automated driving system at level 2or higher, the state of the driver in the automated driving,particularly, the arousal level is monitored, the driver is alerted in acase where the arousal level becomes lower than the acceptable value,and the accident due to lowering in the arousal level has to be therebyprevented. Particularly, a method for precisely estimating the arousallevel of the driver in the automated driving is an important problem.

Meanwhile, in the arousal level estimation device disclosed in JapaneseUnexamined Patent Application Publication No. 6-171391, as describedabove, a method is disclosed which estimates the arousal level of thedriver based on the driving operation by the driver in the automateddriving at level 1 by using the ACC, that is, based on the fluctuationpattern in the steering angle which corresponds to the steeringoperation by the driver. Usually, in a case where the arousal levellowers, the driver may not accurately perform the steering operation andfrequently performs rapid correction of the steering angle. Accordingly,the rapid correction of the steering angle in the steering operation bythe driver may be detected by using a frequency feature of thefluctuation pattern in the steering angle in a unit time. Thus, based onthe detection result, the arousal level of the driver may be estimated.

However, as described above, there is a problem in that the arousallevel estimation device disclosed in Japanese Unexamined PatentApplication Publication No. 6-171391 may not estimate the arousal levelfor the automated driving system at level 2 or higher in which thedriver performs few driving operations such as the steering operation.

Meanwhile, for example, Japanese Unexamined Patent ApplicationPublication No. 2014-181020 or Japanese Unexamined Patent ApplicationPublication No. 2016-38768 discloses a method for estimating the arousallevel of the driver by using physiological information of the driverthat is other information than the driving operation. More specifically,Japanese Unexamined Patent Application Publication No. 2014-181020discloses a method in which the state of the driver such as an eyeopening ratio that is an opening magnitude of an eye of the driver, forexample, is detected from a face image of the driver in the automateddriving, which is acquired by using an in-vehicle camera, and thearousal level of the driver is thereby estimated. Japanese UnexaminedPatent Application Publication No. 2016-38768 discloses a method inwhich the arousal level of the driver is estimated based on thephysiological information of the driver in the automated driving such asheart rate information or brain wave information, for example, which isdetected by using a wearable physiological sensor. Using the methoddisclosed in Japanese Unexamined Patent Application Publication No.2014-181020 or Japanese Unexamined Patent Application Publication No.2016-38768 enables the arousal level of the driver to be estimatedwithout using the driving operation by the driver. That is, it ispossible to estimate the arousal level of the driver even for theautomated driving system at level 2 or higher in which the driverperforms few driving operations.

However, there is a problem in that sufficient precision may not beobtained because in general the physiological information widely variesdue to individual differences. For example, in the method disclosed inJapanese Unexamined Patent Application Publication No. 2014-181020, thesleepiness of the driver is estimated to become higher, that is, thearousal level is estimated to become lower as the eye opening ratio,which is the opening magnitude of the eye of the driver, becomes lower.However, because the driver who habitually opens his/her eye widely andthe driver who does not are present, the arousal level of all thedrivers may not be estimated precisely based on the eye opening ratiothat uses the same reference. Similarly, in the method disclosed inJapanese Unexamined Patent Application Publication No. 2016-38768, thearousal level is estimated by using the heart rate information or thebrain waves of the driver. However, the precision of those pieces ofphysiological information related to the autonomic nervous system andthe central nervous system more widely varies due to individualdifferences than the eye opening ratio. Accordingly, in order toprecisely estimate the arousal level by using the physiologicalinformation of the driver, an estimation method that absorbs variationsof the physiological information due to individual differences isrequested.

Regarding the request, as a method that absorbs variations due toindividual differences among the drivers, a method has been contrivedwhich causes an arousal level estimation model for estimating thearousal level to have a learning function and thereby generates thearousal level estimation model of an individual driver. Specifically, amethod has been contrived in which model learning is successivelyperformed for the arousal level estimation model common to the driversby using teacher data of the individual driver and thereby the arousallevel estimation model for the individual driver is finally generated.

However, there is a problem in that how the teacher data of theindividual driver are generated for the driver in driving may not beknown. At the present time, as the teacher data for the arousal levelestimation model, for example, a subjective evaluation value that isreported by the driver himself/herself or a facial expression evaluationvalue that is obtained from subjective estimation of a facial expressionvideo of the driver by a third party is most frequently used. However,it is very difficult to acquire such data in real time from the driverin driving.

Incidentally, at the present time, it is considered to be technicallyvery difficult for the vehicle to regularly travel in usual urban areasin an automated driving mode at level 2 or higher after the driver rideson the vehicle and starts driving and until the vehicle arrives at adestination and the driver finishes driving. This is because theautomated driving system has to completely recognize the trafficsituation around the vehicle but infrastructure construction forcomplete recognition of the traffic situation around the vehicle is notrealized. Thus, for the time being, it is considered that the travelcontrol of the vehicle is performed by using a manual driving mode atlevel 0 or a safe driving support mode at level 1 in the urban areas andby using the automated driving mode at level 2 or higher in suburbanareas and on highways. That is, for the time being, it is consideredthat the driver performs driving by appropriately switching the manualdriving mode and the automated driving mode in accordance with thesurrounding environment.

Accordingly, as a result of intensive studies, the inventor has foundthat in manual driving, the arousal level estimation model that uses thephysiological information of the driver learns estimation results of thearousal level estimation model, which uses driving information such asthe driving operations by the driver or vehicle behavior, as the teacherdata and the variations in the arousal level estimation model that usesthe physiological information due to individual differences may therebybe absorbed.

An arousal level estimation device according to one aspect of thepresent disclosure is an arousal level estimation device that estimatesarousal level of a driver of a vehicle, the vehicle being capable ofswitching an automated driving mode in which travel control of thevehicle is automatically performed and a manual driving mode in whichthe travel control of the vehicle is performed by the driver. Thearousal level estimation device includes a vehicle behavior detectionunit that detects driving information of the vehicle, a first arousallevel recognition unit that recognizes first arousal level of the driverfrom the driving information detected by the vehicle behavior detectionunit, a physiological information detection unit that detects one ormore pieces of physiological information of the driver, a second arousallevel recognition unit that recognizes second arousal level of thedriver from the one or more pieces of physiological information detectedby the physiological information detection unit, and an arousal levelestimation unit that estimates third arousal level of the driver intraveling of the vehicle from at least one of the first arousal levelrecognized by the first arousal level recognition unit and the secondarousal level recognized by the second arousal level recognition unit.The arousal level estimation unit estimates the third arousal level fromthe first arousal level and the second arousal level in the manualdriving mode and estimates the third arousal level from the secondarousal level in the automated driving mode.

Consequently, the recognition units each of which recognizes the arousallevel of the driver may selectively be used in the manual driving modeand the automated driving mode. Thus, the arousal level of the driver ofthe vehicle that has the automated driving mode and the manual drivingmode may be estimated highly precisely.

Further, in the automated driving mode, the arousal level estimationunit may set the second arousal level as the third arousal level and maythereby estimate the third arousal level from the second arousal level.

Further, in the manual driving mode, the arousal level estimation unitmay assess whether or not reliability of each of the first arousal leveland the second arousal level is equal to or higher than a thresholdvalue, set one of the first arousal level and the second arousal level,whose reliability is equal to or higher than the threshold value, as thethird arousal level, and thereby estimate the third arousal level.

Further, in the manual driving mode, in a case where the reliability ofeach of the first arousal level and the second arousal level is equal toor higher than the threshold value and the first arousal level isdifferent from the second arousal level, the arousal level estimationunit may set the first arousal level as the third arousal level andthereby estimate the third arousal level.

Further, in the manual driving mode, in a case where the first arousallevel is different from the second arousal level, the arousal levelestimation unit may cause the second arousal level recognition unit toperform a learning process such that the second arousal levelrecognition unit outputs the first arousal level as a recognition resultby using teacher data that are generated based on the first arousallevel.

Consequently, in the manual driving mode, an estimation model of thesecond arousal level is learned by using the teacher data generated fromthe first arousal level, and variations in precision due to individualdifferences in the physiological information may thereby be absorbed.Accordingly, it is possible to precisely estimate the arousal level ofthe driver even in the automated driving system at level 2 or higher inwhich the driving information of the driver may not be used.

Further, in the learning process, an arousal level identification modelfor recognizing the second arousal level, which indicates therelationship between one or more pieces of physiological information ofthe driver and the arousal level of the driver, may be updated such thatthe arousal level identification model for recognizing the secondarousal level outputs the first arousal level as an identificationresult.

Further, the second arousal level recognition unit may include aphysiological feature extraction unit that extracts a physiologicalinformation feature related to a physiological state of the driver fromeach of plural pieces of physiological information detected by thephysiological information detection unit and a physiological featureselection unit that selects the physiological information feature, whichis highly correlated with the teacher data generated from the firstarousal level, among plural physiological information features extractedby the physiological feature extraction unit, and the arousal levelestimation unit may cause the second arousal level recognition unit toperform the learning process such that the second arousal levelrecognition unit outputs the first arousal level as a recognition resultby using the physiological information feature that is selected by thephysiological feature selection unit as the teacher data.

Further, the physiological information may be information that indicatesa heart rate fluctuation of the driver.

Further, the physiological information may be a face image in which aface of the driver appears.

Further, the physiological information may be information that indicatesbody movement of the driver.

Further, the driving information may be information that indicates asteering angle of the vehicle.

Further, the driving information may be information that indicatespositions of an accelerator pedal and a brake pedal of the vehicle.

Further, the driving information may be information that indicatesacceleration of the vehicle.

Further, an arousal level estimation method according to one aspect ofthe present disclosure is an arousal level estimation method forestimating arousal level of a driver of a vehicle, the vehicle beingcapable of switching an automated driving mode in which travel controlof the vehicle is automatically performed and a manual driving mode inwhich the travel control of the vehicle is performed by the driver. Thearousal level estimation method includes a vehicle behavior detectionstep of detecting driving information of the vehicle, a first arousallevel recognition step of recognizing first arousal level of the driverfrom the driving information detected in the vehicle behavior detectionstep, a physiological information detection step of detectingphysiological information of the driver, a second arousal levelrecognition step of recognizing second arousal level of the driver fromthe physiological information detected in the physiological informationdetection step, and an arousal level estimation step of estimating thirdarousal level of the driver in traveling of the vehicle from at leastone of the first arousal level recognized in the first arousal levelrecognition step and the second arousal level recognized in the secondarousal level recognition step. In the arousal level estimation step,the third arousal level is estimated based on the first arousal leveland the second arousal level in the manual driving mode, and the thirdarousal level is estimated based on the second arousal level in theautomated driving mode.

Note that the present disclosure may be realized not only as a devicebut also realized as an integrated circuit that includes a processingsection which is included in such a device, a method including stepsusing the processing section which configures the device, a program thatcauses a computer to execute the steps, and information, data, orsignals that indicate the program. Further, the program, information,data, and signals may be delivered via a recording medium such as aCD-ROM or a communication medium such as the Internet.

Embodiments of the present disclosure will hereinafter be described withreference to drawings.

Note that all the embodiments described below illustrate general orspecific examples. Values, shapes, materials, configuration elements,arrangement positions or connection manners of configuration elements,steps, orders of steps, and so forth that are described in the followingembodiments are examples and are not intended to limit the presentdisclosure. Further, the configuration elements that are not describedin the independent claims which provide the most superordinate conceptsamong the configuration elements in the following embodiments will bedescribed as arbitrary configuration elements.

Embodiment

A driving support device according to an embodiment will hereinafter bedescribed.

[1. Configuration of Driving Support Device 1]

FIG. 1 is a block diagram that illustrates one example of aconfiguration of a driving support device 1 in this embodiment.

The driving support device 1 is a device that is installed in a vehicle,estimates the arousal level of a driver in driving, alerts the driver ina case where the estimated arousal level is lower than a prescribedvalue, and thereby inhibits drowsy driving by the driver. In thisembodiment, a description will be made in the following on an assumptionthat the vehicle is capable of switching an automated driving mode inwhich travel control of the vehicle is automatically performed and amanual driving mode in which the travel control of the vehicle isperformed by a driving operation by the driver.

As illustrated in FIG. 1, the driving support device 1 includes adriving operation unit 2, a surrounding environment recognition unit 3,a subject vehicle position detection unit 4, a target travel statedecision unit 5, a driving mode selection unit 6, a vehicle control unit7, a notification unit 8, and an arousal level estimation device 10.

<Driving Operation Unit 2>

FIG. 2A is a block diagram that illustrates one example of a specificconfiguration of the driving operation unit 2 illustrated in FIG. 1.

The driving operation unit 2 is operated in order for the driver todrive the vehicle. For example, as illustrated in FIG. 2A, the drivingoperation unit 2 includes a steering wheel 21, an accelerator pedal 22,a brake pedal 23, a steering lever 24 such as a shift lever, and adriving mode selection switch 25 for selecting a driving mode. Note thatthe pedals included in the driving operation unit 2 are not limited tothe accelerator pedal 22 and the brake pedal 23. Further, the drivingoperation unit 2 may include other components than the above.

<Surrounding Environment Recognition Unit 3>

FIG. 2B is a block diagram that illustrates one example of a specificconfiguration of the surrounding environment recognition unit 3illustrated in FIG. 1.

The surrounding environment recognition unit 3 includes plural sensorsand plural recognition units and recognizes a surrounding environment ofthe subject vehicle.

The plural sensors detect various kinds of information for monitoring anoutside situation of the subject vehicle. For example, as illustrated inFIG. 2B, the plural sensors include a camera 31 and a radar 32. At leastone or more cameras 31 are arranged on an external surface of thevehicle, for example, and image an outside environment of the vehiclesuch as a lane, a road surface, and a surrounding obstacle. The radar 32is a millimeter-wave radar or a laser radar that is arranged in a frontportion or a rear portion of the vehicle, for example, and measures thedistances to and the positions of a vehicle or an obstacle that ispresent around the subject vehicle, and so forth.

The plural recognition units recognize the surrounding environment ofthe subject vehicle based on the information detected from each of thesensors. For example, as illustrated in FIG. 2B, the plural recognitionunits include a road shape recognition unit 33, a road surface staterecognition unit 34, and a surrounding object recognition unit 35. Theroad shape recognition unit 33 recognizes a road shape around thesubject vehicle. The road shape recognition unit 33 recognizes the lanesuch as a white line on a road from a photographed image by the camera31 and detects the width, the curvature, and so forth of the lane. Theroad surface state recognition unit 34 recognizes a road surface statein front of the subject vehicle. The road surface state recognition unit34 recognizes the road surface state such as a snow accumulation orblack ice from a photographed image by the camera 31. The surroundingobject recognition unit 35 recognizes an object that is present aroundthe subject vehicle. The surrounding object recognition unit 35recognizes a pedestrian, a bicycle, a motorcycle, or another surroundingvehicle by using photographed images by the camera 31 and radarinformation by the radar 32 and detects the size, position, speed,movement direction, or the like of the recognized object.

<Subject Vehicle Position Detection Unit 4>

FIG. 2C is a block diagram that illustrates one example of a specificconfiguration of the subject vehicle position detection unit 4illustrated in FIG. 1.

As illustrated in FIG. 2C, the subject vehicle position detection unit 4includes a global positioning system (GPS) 41, a vehicle travel positioncalculation unit 42, and a map data storage unit 43 and detects theposition of the subject vehicle on the map. The GPS 41 receives GPSinformation that indicates the position of the subject vehicle from GPSsatellites. The map data storage unit 43 in advance stores map data. Thevehicle travel position calculation unit 42 calculates the travelposition of the subject vehicle on the map data that are stored in themap data storage unit 43 based on the received GPS information.

<Target Travel State Decision Unit 5>

The target travel state decision unit 5 decides a target travel statethat is a travel state of the subject vehicle to be a target. In theexample illustrated in FIG. 1, the target travel state decision unit 5calculates information of the target travel state based on presentsurrounding environment information that is obtained from thesurrounding environment recognition unit 3, a present subject vehicleposition that is obtained from the subject vehicle position detectionunit 4, and present subject vehicle behavior that is obtained from avehicle behavior detection unit 11. Note that information of the travelstate includes information related to travel of the vehicle such as theposition of the vehicle or the speed or orientation of the vehicle inthe position. The target travel state may be calculated by a method inrelated art, and a description thereof will thus not be made here.

<Driving Mode Selection Unit 6>

The driving mode selection unit 6 switches the driving mode of thevehicle to either one of the automated driving mode and the manualdriving mode. In the example illustrated in FIG. 1, the driving modeselection unit 6 switches the driving mode to either one of theautomated driving mode and the manual driving mode in accordance withthe operation of the driving mode selection switch 25 of the drivingoperation unit 2 by the driver.

<Vehicle Control Unit 7>

The vehicle control unit 7 is configured with a CPU or the like thatrealizes a prescribed function in cooperation with software, forexample, and performs the travel control of the vehicle based on thepresently-selected driving mode. For example, in the automated drivingmode, the vehicle control unit 7 controls various actuators foracceleration, steering, and braking of the subject vehicle, an ECU, andso forth based on the target travel state that is obtained from thetarget travel state decision unit 5. Further, for example, in the manualdriving mode, the vehicle control unit 7 controls the various actuatorsfor acceleration, steering, and braking of the subject vehicle, the ECU,and so forth in accordance with the operation of the driving operationunit 2 by the driver.

<Notification Unit 8>

The notification unit 8 is configured with various in-vehicle displayssuch as a car navigation system, a speaker, a driver seat that has avibration actuator built therein, or the like, for example. Thenotification unit 8 provides notification of the driving mode that isselected by the driver or information that indicates the present vehicleposition. Further, the notification unit 8 notifies the driver ofpredetermined information in accordance with the arousal level that isestimated by an arousal level estimation unit 15. For example, thenotification unit 8 provides notification of an alert or a warning thatsuppresses the sleepiness of the driver in a case where a determinationis made that the arousal level of the driver becomes lower than aprescribed value, that is, the sleepiness equal to or higher than aprescribed value occurs to the driver.

<Arousal Level Estimation Device 10>

The arousal level estimation device 10 is one example of an arousallevel estimation device in the present disclosure and estimates thearousal level of the driver in driving of the vehicle. A specificconfiguration and so forth of the arousal level estimation device 10will hereinafter be described with reference to the drawings.

[Configuration of Arousal Level Estimation Device 10]

FIG. 3 is a block diagram that illustrates examples of specificconfigurations of the vehicle behavior detection unit 11 and a firstarousal level recognition unit 12, which are illustrated in FIG. 1. FIG.4 is a block diagram that illustrates examples of specificconfigurations of a physiological information detection unit 16 and asecond arousal level recognition unit 17, which are illustrated in FIG.1.

As illustrated in FIG. 1, the arousal level estimation device 10includes the vehicle behavior detection unit 11, the first arousal levelrecognition unit 12, the arousal level estimation unit 15, thephysiological information detection unit 16, and the second arousallevel recognition unit 17. Note that the arousal level (sleepiness) ofthe driver that is estimated by the arousal level estimation device 10is decided as any one of five levels of level 1 to level 5, for example.For example, level 1 indicates “not sleepy state”, and level 2 indicates“slightly sleepy state”. Level 3 indicates “sleepy state”, and level 4indicates “fairly sleepy state”. Level 5 indicates “very sleepy state”.

<Vehicle Behavior Detection Unit 11>

The vehicle behavior detection unit 11 detects driving information. Inthis embodiment, the vehicle behavior detection unit 11 is configuredwith a sensor group for detecting the driving information that indicatesthe driving operation by the driver and the behavior of the vehicle.This sensor group includes a sensor that detects the driving operationby the driver and a sensor that detects the behavior of the vehicle.

For example, as illustrated in FIG. 3, the sensor that detects thedriving operation by the driver includes a steering angle sensor 111, anaccelerator pedal position sensor 112, and a brake pedal position sensor113. The steering angle sensor 111 detects the rotational angle of thesteering wheel 21 in the driving operation unit 2, that is, the steeringangle. The accelerator pedal position sensor 112 detects the position ofthe accelerator pedal 22 or the brake pedal 23. The brake pedal positionsensor 113 detects the position of the brake pedal 23.

Further, the sensor that detects the behavior of the vehicle includes avehicle speed sensor 114, an acceleration sensor 115, and a yaw ratesensor 116. The vehicle speed sensor 114 detects the speed of thevehicle. The acceleration sensor 115 detects the acceleration in thefront-rear direction of the vehicle and the acceleration in theleft-right direction of the vehicle. The yaw rate sensor 116 detects therotational angle (yaw rate) with respect to the perpendicular directionof the vehicle.

<First Arousal Level Recognition Unit 12>

The first arousal level recognition unit 12 recognizes first arousallevel of the driver from the driving information that is detected by thevehicle behavior detection unit 11. Here, for example, the drivinginformation may be information that indicates the steering angle of thevehicle, may be information that indicates the positions of theaccelerator pedal 22 and the brake pedal 23 of the vehicle, or may beinformation that indicates the acceleration of the vehicle. Here, forexample, the driving information may be information that indicates thespeed of the vehicle or may be information that indicates the yaw rate.In this embodiment, as illustrated in FIG. 3, the first arousal levelrecognition unit 12 includes a driving information feature extractionunit 121 and a first arousal level identification unit 122 andrecognizes the first arousal level of the driver by using thesurrounding environment of the subject vehicle that is recognized by thesurrounding environment recognition unit 3 and the driving informationthat is detected by the vehicle behavior detection unit 11.

<<Driving Information Feature Extraction Unit 121>>

The driving information feature extraction unit 121 extracts a drivinginformation feature from the surrounding environment of the subjectvehicle that is recognized by the surrounding environment recognitionunit 3 and the driving information that is detected by the vehiclebehavior detection unit 11.

Here, the surrounding environment of the subject vehicle is informationthat indicates the vehicular gap between a preceding vehicle and thesubject vehicle or the distance between the subject vehicle and the lanesuch as the white line of the road, for example. Further, as describedabove, the driving information is information that is detected by thesensor group included in the vehicle behavior detection unit 11 and thatindicates the driving operation by the driver such as the steering angleof the steering wheel 21 or the behavior of the subject vehicle such asthe speed of the vehicle, for example. Further, the driving informationfeature is a feature for identifying the first arousal level of thedriver, the feature being related to a driving activity of the driver.Specifically, for example, the driving information features arestatistical features such as average values and standard deviations thatare obtained from fluctuation patterns in the steering angle of thesteering wheel 21 which is operated by the driver and in the positionsof the accelerator pedal 22 and the brake pedal 23 and fluctuationpatterns in the vehicular gap from the preceding vehicle, the lateraldisplacement amount of the vehicle, the vehicle speed, the accelerationsof the vehicle in the front-rear and left-right directions, the yawrate, and so forth, in a prescribed unit time such as one minute.

In this embodiment, the driving information feature extraction unit 121includes one or more feature extraction units and extracts at least oneor more features that are related to the driving activity of the driveras the driving information features. For example, in the exampleillustrated in FIG. 3, the driving information feature extraction unit121 has a vehicular gap feature extraction unit 131, a vehicle lateraldisplacement amount feature extraction unit 132, a steering anglefeature extraction unit 133, an accelerator pedal position featureextraction unit 134, a brake pedal position feature extraction unit 135,a vehicle speed feature extraction unit 136, a vehicle front-reardirection acceleration feature extraction unit 137, a vehicle left-rightdirection acceleration feature extraction unit 138, and a yaw ratefeature extraction unit 139.

The vehicular gap feature extraction unit 131 extracts a vehicular gapfeature, which is the average value, standard deviation, or the like ofthe vehicular gap in a prescribed unit time, as the driving informationfeature by using the vehicular gap between the preceding vehicle and thesubject vehicle that is obtained from the surrounding environmentrecognition unit 3.

The vehicle lateral displacement amount feature extraction unit 132acquires, from the surrounding environment recognition unit 3,information of the road shape such as the width, the curvature, or thelike of the lane and information that indicates the distance between thevehicle and the lane such as the distance from the white line of theroad. The vehicle lateral displacement amount feature extraction unit132 calculates the displacement amount of the vehicle from a center lineof the road (hereinafter referred to as vehicle lateral displacementamount) based on the information of the road shape and the distance thatis obtained from the surrounding environment recognition unit 3. Then,the vehicle lateral displacement amount feature extraction unit 132extracts a vehicle lateral displacement feature, which is the averagevalue, standard deviation, or the like of the vehicle lateraldisplacement amount in a prescribed unit time which is calculated byusing the pieces of information and the displacement amount, as thedriving information feature. Note that the vehicle lateral displacementamount may not be a simple displacement amount from the center line butmay be the displacement amount from the travel position that is set as atarget at the time after a prescribed unit time by the driver. Further,for example, as disclosed in Japanese Unexamined Patent ApplicationPublication No. 2015-219771, a target travel position of the driver maybe calculated based on the steering angle of the steering wheel 21, thespeed of the vehicle, or the yaw rate, which is obtained from thevehicle behavior detection unit 11.

The steering angle feature extraction unit 133 extracts a steering anglefeature, which is the average value or standard deviation of thedisplacement amount of the steering angle in a prescribed unit time, asthe driving information feature by using the steering angle of thesteering wheel 21 that is obtained from the steering angle sensor 111.Note that the steering angle feature extraction unit 133 may extract thesteering angle feature, which is a frequency feature or the like of thedisplacement amount of the steering angle, as the driving informationfeature.

The accelerator pedal position feature extraction unit 134 extracts anaccelerator pedal position feature, which is the average value, standarddeviation, or the like of the displacement amount of the position of theaccelerator pedal in a prescribed unit time, as the driving informationfeature by using the position of the accelerator pedal 22 that isobtained from the accelerator pedal position sensor 112.

The brake pedal position feature extraction unit 135 extracts a brakepedal position feature, which is calculated by using the position of thebrake pedal 23 which is obtained from the brake pedal position sensor113, as the driving information feature. Here, the brake pedal positionfeature is the average value or standard deviation of the displacementamount of the position of the brake pedal in a prescribed unit time, thefrequency of brake operations, or the like.

The vehicle speed feature extraction unit 136 extracts a vehicle speedfeature, which is the average value, standard deviation, or the like ofthe displacement amount of the vehicle speed in a prescribed unit time,as the driving information feature by using the vehicle speed that isobtained from the vehicle speed sensor 114.

The vehicle front-rear direction acceleration feature extraction unit137 calculates the acceleration of the vehicle in the front-reardirection of the vehicle accelerations that are obtained from theacceleration sensor 115. Further, the vehicle front-rear directionacceleration feature extraction unit 137 extracts a vehicle front-reardirection acceleration feature, which is the average value, standarddeviation, or the like of the displacement amount of the acceleration ofthe vehicle in the front-rear direction in a prescribed unit time, asthe driving information feature by using the calculated acceleration ofthe vehicle in the front-rear direction.

The vehicle left-right direction acceleration feature extraction unit138 calculates the acceleration of the vehicle in the left-rightdirection of the vehicle accelerations that are obtained from theacceleration sensor 115. Further, the vehicle left-right directionacceleration feature extraction unit 138 extracts a vehicle left-rightdirection acceleration feature, which is the average value, standarddeviation, or the like of the displacement amount of the acceleration ofthe vehicle in the left-right direction in a prescribed unit time, asthe driving information feature by using the calculated acceleration ofthe vehicle in the left-right direction.

The yaw rate feature extraction unit 139 extracts a yaw rate feature,which is the average value, standard deviation, or the like of thedisplacement amount of the yaw rate in a prescribed unit time, as thedriving information feature by using the yaw rate that is obtained fromthe yaw rate sensor 116.

<<First Arousal level Identification Unit 122>>

The first arousal level identification unit 122 identifies the firstarousal level of the driver by using the driving information featuresthat are extracted by the driving information feature extraction unit121.

In this embodiment, the first arousal level identification unit 122identifies the arousal level of the driver by using at least one or morefeatures that are obtained from one or more feature extraction units ofthe driving information feature extraction unit 121 and outputsidentification results as the first arousal level. The identificationresults include the arousal level that indicates any of level 1 to level5 and the reliability of the arousal level, for example.

Here, a specific description will be made about an identification methodof the first arousal level that is performed by the first arousal levelidentification unit 122.

It has been known that the driving operation of the vehicle by thedriver becomes monotonous in a case where the arousal level of thedriver lowers (the sleepiness becomes high) in driving. For example, theoperations of the steering wheel 21, the accelerator pedal 22, or thebrake pedal 23 by the driver lessen in a case where the sleepinessbecomes high compared to a time before the sleepiness becomes high.Further, in a case where the sleepiness of the driver becomes high, thevehicle in driving wobbles, and the fluctuation in the lateraldisplacement amount of the vehicle, the vehicular gap, or the like tendsto become large. For example, in a case where the sleepiness of thedriver temporarily becomes high, a case occurs where the operations ofthe steering wheel 21 decrease and the vehicle wobbles hard. However, ina moment when the vehicle wobbles hard, the driver often awakestemporarily and takes an activity to rapidly operate the steering wheel21 in order to correct the travel state of the vehicle to a right travelstate. Further, in such a case, the fluctuation pattern in thedisplacement amount of the steering angle of the steering wheel 21 orthe lateral displacement amount of the vehicle exhibit a feature patternin which the displacement amount becomes comparatively small for acertain time and the displacement amount thereafter rapidly and largelychanges due to a rapid correction, for example.

Accordingly, a specific fluctuation pattern in each of the drivinginformation features due to the sleepiness is detected, and thesleepiness of the driver may thereby be detected (identified). Morespecifically, an arousal level identification model which may associatesuch a specific fluctuation pattern in each motion feature due to thesleepiness with the arousal level in such a case is generated, thegenerated arousal level identification model is used, and the sleepinessof the driver may thereby be identified.

In this embodiment, the first arousal level identification unit 122 usesthe arousal level identification model that is generated in such amanner, thereby identifies the first arousal level from drivinginformation features, and outputs the first arousal level asidentification results to the arousal level estimation unit 15. Notethat, as described above, the identification results include the arousallevel that indicates any of level 1 to level 5 and the reliability thatindicates the certainty of the arousal level in addition, for example.For example, as the identification results, arousal level: 3,reliability: 0.8, and so forth are output. Further, the generatedarousal level identification model uses at least one or more drivinginformation features, which are selected from the driving informationfeatures obtained from the driving information feature extraction unit121, as input data and outputs the arousal level of the driver. Thearousal level identification model may be generated by using a machinelearning algorithm in related art such as a neural network, a supportvector machine, or a random forest.

<Physiological information Detection Unit 16>

The physiological information detection unit 16 detects one or morepieces of physiological information of the driver. In this embodiment,the physiological information detection unit 16 is configured with asensor group for detecting the physiological information of the driver.For example, as illustrated in FIG. 4, the sensor group has anin-vehicle camera 161, a microphone 162, a heart rate sensor 163, ablood pressure sensor 164, and a body movement sensor 165. Note that thesensor group may further have a respiration sensor (not illustrated)that detects a respiration fluctuation of the driver, a skin temperaturesensor (not illustrated) that detects the skin temperature of thedriver, and so forth.

The in-vehicle camera 161 is arranged in the vehicle and images aninternal portion of the vehicle. Then, the in-vehicle camera 161generates image data that indicate a photographed image. For example,the in-vehicle camera 161 photographs the driver in driving. Morespecifically, the in-vehicle camera 161 is arranged to be capable ofphotographing the vicinity of a driver seat in the vehicle andphotographs the driver on the driver seat.

The microphone 162 collects sound in the vehicle. More specifically, themicrophone 162 is arranged in the vehicle to collect surrounding sound.Then, the microphone 162 generates data (audio data) that indicate thecollected sound.

The heart rate sensor 163 detects a heart rate fluctuation of thedriver. More specifically, the heart rate sensor 163 measures the heartrate of the driver and generates a sensor signal of the measurementresult. Note that the heart rate sensor 163 may be a contact type sensordevice that is attached to the body such as the earlobe or may be acontactless type sensor device such as a camera that extracts the changein the complexion which corresponds to a pulse wave. Further, the heartrate sensor 163 may be substituted by a pulse wave sensor and detect theheart rate fluctuation of the driver.

The blood pressure sensor 164 measures the blood pressure of the driverand generates a sensor signal of the measurement result. Here, the bloodpressure sensor 164 is configured with a wearable device, for example,and is in advance attached to a person in charge who is to performdriving in the manual driving mode, that is, the driver.

The body movement sensor 165 detects the body movement such as thechange in the posture of the driver. More specifically, the bodymovement sensor 165 is configured with a load sensor that is arranged inan internal portion of a backrest or a seating surface of the driverseat, for example. The body movement sensor 165 senses the change in theposture of a person on the driver seat and generates a sensor signal ofthe sensing result. Note that the body movement sensor 165 may beconfigured with an acceleration sensor, an angular velocity sensor, orthe like.

<Second Arousal level Recognition Unit 17>

The second arousal level recognition unit 17 recognizes second arousallevel of the driver from one or more pieces of physiological informationthat are detected by the physiological information detection unit 16.Here, the physiological information may be information that indicatesthe heart rate fluctuation of the driver, may be a face image in whichthe face of the driver appears, or may be information that indicates thebody movement of the driver, for example. Further, the physiologicalinformation may be information that indicates the respiration of thedriver or may be information that indicates the blood pressure of thedriver, for example. In this embodiment, as illustrated in FIG. 4, thesecond arousal level recognition unit 17 includes a physiologicalfeature extraction unit 171 and a second arousal level identificationunit 172. Note that in a case where the second arousal level recognitionunit 17 receives the teacher data from a teacher data generation unit152, which will be described later, the second arousal level recognitionunit 17 performs learning for the second arousal level identificationunit 172 based on the teacher data.

<<Physiological Feature Extraction Unit 171>>

The physiological feature extraction unit 171 extracts a physiologicalinformation feature related to the physiological state of the driverfrom each of plural pieces of physiological information that aredetected by the physiological information detection unit 16.

Here, the physiological information feature is a feature for identifyingthe arousal level of the driver and a feature that is related tophysiological information of the driver. The physiological informationis an index that indicates a physiological state of a person such assleepiness or fatigue and includes visual system physiologicalinformation, autonomic nervous system physiological information,skeletal system physiological information, or the like, for example. Thevisual system physiological information is a statistical feature such asthe average value, standard deviation, or the like that is obtained fromthe fluctuation pattern in the eye opening ratio of the driver whichindicates the opening degree of the eye of the driver, the frequency ortime of eye blink of the driver, the position of the head of the driver,or the like in a prescribed unit time such as one minute, for example.Further, the autonomic nervous system physiological information is astatistical feature that is obtained from the fluctuation patterns inthe respiration rate, the depth of respiration, the heart rate, theblood pressure, or the like in a prescribed unit time. Further, theskeletal system physiological information is a statistical feature thatis obtained from the fluctuation pattern in the position of the centerof gravity of the body in a prescribed unit time.

In this embodiment, the physiological feature extraction unit 171includes one or more feature extraction units and extracts at least oneor more features that indicate the physiological state of the driver.For example, as illustrated in FIG. 4, the physiological featureextraction unit 171 has a face image extraction unit 181, an eye openingratio detection unit 182, an eye opening ratio feature extraction unit183, an eye blink detection unit 184, an eye blink feature extractionunit 185, a head position detection unit 186, a head position featureextraction unit 187, a respiratory sound detection unit 188, arespiration feature extraction unit 189, a heart rate fluctuationdetection unit 190, a heart rate fluctuation feature extraction unit191, a blood pressure fluctuation detection unit 192, a blood pressurefluctuation feature extraction unit 193, a body movement detection unit194, and a body movement feature extraction unit 195.

For example, the physiological feature extraction unit 171 uses the faceimage extraction unit 181 to the head position feature extraction unit187 and may thereby extract an eye opening ratio feature, an eye blinkfeature, or a head position feature, which is related to the visualsystem physiological information of the driver, from an image which isphotographed by the in-vehicle camera 161 and includes the driver, asthe physiological information feature. Specifically, the face imageextraction unit 181 consecutively acquires the image data from thein-vehicle camera 161 and extracts a face image region of the driverfrom the image data. The eye opening ratio detection unit 182 extractsthe image region of the eye from the face image region extracted by theface image extraction unit 181, thereafter detects the upper and lowereyelids, and detects the eye opening ratio that indicates the openingdegree of the eyelids, that is, the eye from the shapes of the eyelids.The eye opening ratio feature extraction unit 183 calculates an eyeopening ratio feature, which is the average value, standard deviation,maximum value, minimum value, or the like of the eye opening ratio in aprescribed unit time, from the fluctuation pattern in the eye openingratio that is detected by the eye opening ratio detection unit 182.Further, the eye blink detection unit 184 extracts the image region ofthe eye from the face image region extracted by the face imageextraction unit 181, thereafter detects the upper and lower eyelids, anddetects a blink (eye blink) from the movement of the upper and lowereyelids. The eye blink feature extraction unit 185 calculates an eyeblink feature, which is the average value, standard deviation, or thelike of the eye blink frequency or the eye blink time in a prescribedunit time. In addition, the head position detection unit 186 detects thefluctuation pattern in the head position of the driver based on theposition in the image in the face image region extracted by the faceimage extraction unit 181. The head position feature extraction unit 187calculates a head position feature, which is the average value, standarddeviation, maximum value, minimum value, or the like of the headposition in a prescribed unit time, from the fluctuation pattern in thehead position.

Further, for example, the physiological feature extraction unit 171 usesthe respiratory sound detection unit 188 and the respiration featureextraction unit 189 and may thereby extract a respiration feature, whichis one of pieces of the autonomic nervous system physiologicalinformation of the driver, as the physiological information feature fromcollected sound data from the microphone 162 in the vehicle.Specifically, the respiratory sound detection unit 188 detects arespiratory sound pattern of the driver from the collected sound datafrom the microphone 162. The respiration feature extraction unit 189calculates a respiration feature, which is the average value, standarddeviation, maximum value, or the like of the respiration rate or thedepth of respiration in a prescribed unit time, from the respiratorysound pattern that is detected by the respiratory sound detection unit188.

Further, for example, the physiological feature extraction unit 171 usesthe heart rate fluctuation detection unit 190 and the heart ratefluctuation feature extraction unit 191 and may thereby extract a heartrate fluctuation feature, which is one of pieces of the autonomicnervous system physiological information of the driver, as thephysiological information feature from the heart rate sensor 163.Specifically, the heart rate fluctuation detection unit 190 detects an Rwave that has the highest peak from an electrocardiographic waveformobtained from the heart rate sensor 163 and thereafter detects thefluctuation in a heartbeat interval that is the interval between the Rwaves (RR interval (RRI)) as a heart rate fluctuation pattern. The heartrate fluctuation feature extraction unit 191 calculates a heart ratefluctuation feature, which is the average value, standard deviation, orthe like of the heartbeat interval (RRI) in a prescribed unit time, fromthe detected heart rate fluctuation pattern. Note that, as the heartrate fluctuation feature, after the frequency spectrum of the heart ratefluctuation pattern in a prescribed unit time is obtained, the ratio ofpower between a low frequency component (LF component) and a highfrequency component (HF component) or the ratio between the LF componentand the HF component (LF/HF ratio) may be used.

Further, for example, the physiological feature extraction unit 171 usesthe blood pressure fluctuation detection unit 192 and the blood pressurefluctuation feature extraction unit 193 and may thereby extract a bloodpressure fluctuation feature, which is one of pieces of the autonomicnervous system physiological information of the driver, as thephysiological information feature from the blood pressure sensor 164.Specifically, the blood pressure fluctuation detection unit 192 detectsa blood pressure fluctuation pattern of the driver from data obtainedfrom the blood pressure sensor 164. The blood pressure fluctuationfeature extraction unit 193 calculates a blood pressure fluctuationfeature, which is the average value, standard deviation, maximum value,minimum value, or the like of the blood pressure in a prescribed unittime, from the detected blood pressure fluctuation pattern.

Further, for example, the physiological feature extraction unit 171 usesthe body movement detection unit 194 and the body movement featureextraction unit 195 and may thereby extract a body movement feature,which is one of pieces of the skeletal system physiological informationof the driver, as the physiological information feature from the bodymovement sensor 165. Specifically, the body movement detection unit 194detects a body movement pattern that indicates the body movement of thedriver from data obtained from the body movement sensor 165. The bodymovement feature extraction unit 195 calculates a body movement feature,which is the average value, standard deviation, maximum value, minimumvalue, or the like of the body movement frequency, the fluctuationamount of the position of the center of gravity of the body (bodymovement fluctuation amount), or the like in a prescribed unit time,from the detected body movement pattern.

<<Second Arousal level Identification Unit 172>>

The second arousal level identification unit 172 identifies the secondarousal level of the driver by using the physiological informationfeatures that are extracted by the physiological feature extraction unit171.

In this embodiment, the second arousal level identification unit 172identifies the arousal level of the driver by using at least one or morefeatures that are obtained from one or more feature extraction units ofthe physiological feature extraction unit 171 and outputs identificationresults as the second arousal level. Note that the identificationresults include the arousal level that indicates any of level 1 to level5 and the reliability of the arousal level, for example.

Here, a specific description will be made about an identification methodof the second arousal level that is performed by the second arousallevel identification unit 172.

It has been known that in a case where the arousal level of the driverlowers (the sleepiness becomes high) in driving, the visual systemphysiological information such as the eye opening ratio and the eyeblink of the driver, the autonomic nervous system physiologicalinformation such as the heart rate and the pulse wave, and the skeletalsystem physiological information such as the body movement exhibitparticular tendencies. First, a description will be made about theparticular tendencies due to the sleepiness in relation to the visualsystem physiological information. Because the eyelids tend to close in acase where the sleepiness of the driver becomes high, a tendency isexhibited in which the eye opening ratio which indicates the openingdegree of the eyelids becomes low. Further, a tendency is exhibited inwhich in a case where the sleepiness of the driver becomes high, the eyeblink time of closing the eyelids for one eye blink becomes long, andthe eye blink frequency in a prescribed unit time decreases. Further,because the head movement become unstable in a case where the sleepinessof the driver becomes high, a tendency is exhibited in which the headposition unstably fluctuates.

Next, a description will be made about the particular tendencies due tothe sleepiness in relation to the autonomic nervous system physiologicalinformation. Because one respiratory stroke becomes deep and therespiratory interval becomes long in a case where the sleepiness of thedriver becomes high, a tendency is exhibited in which the respirationrate in a prescribed unit time decreases. Further, because the heartrate lowers in a case where the sleepiness of the driver becomes high, atendency is exhibited in which the heartbeat interval (RRI) in aprescribed unit time becomes long. Further, as for the frequencyspectrum of the heart rate fluctuation pattern (RRI), it has been knownthat the high frequency (HF: 0.15 to 0.4 Hz) component is low in anactive condition and the HF component becomes high in a relaxedcondition. Thus, the ratio between the low frequency (LF: 0.04 to 0.15Hz) component and the HF component (LF/HF) is often used as an index ofsleepiness. That is, a tendency is exhibited in which the LF/HF ratio ofthe driver becomes low in a case where the sleepiness of the driverbecomes high. Further, a tendency is exhibited in which the bloodpressure temporarily rises because the driver resists the sleepiness ina case where the driver starts feeling the sleepiness but the bloodpressure gradually falls in a case where the sleepiness of the driverfurther becomes high.

Finally, a description will be made about the particular tendencies dueto the sleepiness in relation to the skeletal system physiologicalinformation. A tendency is exhibited in which the body movementfrequency increases in order to resist the sleepiness and the bodymovement fluctuation amount also increases in a case where the driverstarts feeling the sleepiness. Further, a tendency is exhibited in whichin a case where the sleepiness of the driver further becomes high, thedriver may not resist the sleepiness, and the body movement frequencyand the body movement fluctuation amount decrease.

In such a manner, each of the physiological information features due tothe sleepiness exhibits a particular tendency, that is, a specificfluctuation pattern. Accordingly, the specific fluctuation pattern ineach of the physiological information features due to the sleepiness isdetected, and the sleepiness of the driver may thereby be detected(identified). More specifically, an arousal level identification modelwhich may associate such a specific fluctuation pattern in each of thephysiological information features due to the sleepiness with thearousal level in such a case is generated, the generated arousal levelidentification model is used, and the sleepiness of the driver maythereby be identified.

In this embodiment, the second arousal level identification unit 172uses the arousal level identification model that is generated in such amanner, thereby identifies the second arousal level from physiologicalinformation features, and outputs the second arousal level asidentification results to the arousal level estimation unit 15. Notethat, as described above, the identification results include the arousallevel that indicates any of level 1 to level 5 and the reliability thatindicates the certainty of the arousal level in addition, for example.For example, as the identification results, arousal level: 3,reliability: 0.8, and so forth are output. Further, the generatedarousal level identification model uses at least one or morephysiological information features, which are selected from thephysiological information features obtained from the physiologicalfeature extraction unit 171, as input data and outputs the arousal levelof the driver. The arousal level identification model may be generatedby using the machine learning algorithm in related art such as theneural network, the support vector machine, or the random forest.

Note that the second arousal level identification unit 172 performslearning based on the teacher data generated by the teacher datageneration unit 152, which will be described later. More specifically,in the second arousal level identification unit 172, in a case where thefirst arousal level is different from the second arousal level in themanual driving mode, a learning process is performed by using theteacher data that are generated by the teacher data generation unit 152based on the first arousal level. That is, in the learning, model updateis performed such that the arousal level identification model of thesecond arousal level identification unit 172, which indicates therelationship between one or more pieces of physiological information ofthe driver and the arousal level of the driver, outputs the firstarousal level as the identification result from the teacher data.

In this embodiment, in a case where the second arousal levelidentification unit 172 acquires the teacher data from the teacher datageneration unit 152, the second arousal level identification unit 172performs learning for the arousal level identification model that isused for identifying the second arousal level by using the teacher data.Note that, as a specific learning method, for example, a backpropagationmethod or the like may be used in a case where the arousal levelidentification model is a hierarchical neural network.

<Arousal level Estimation Unit 15>

The arousal level estimation unit 15 estimates third arousal level ofthe driver in driving of the vehicle from at least one of the firstarousal level that is recognized by the first arousal level recognitionunit 12 and the second arousal level that is recognized by the secondarousal level recognition unit 17.

Here, the arousal level estimation unit 15 may estimate the thirdarousal level from the first arousal level and the second arousal levelin the manual driving mode and estimate the third arousal level from thesecond arousal level in the automated driving mode. More specifically,the arousal level estimation unit 15 may set the second arousal level asthe third arousal level and thereby estimate the third arousal levelfrom the second arousal level in the automated driving mode. Further, inthe manual driving mode, the arousal level estimation unit 15 may assesswhether or not the reliability of each of the first arousal level andthe second arousal level is a threshold value or higher, set either oneof the first arousal level and the second arousal level, whosereliability is the threshold value or higher, as the third arousallevel, and thereby estimate the third arousal level. More specifically,in the manual driving mode, in a case where the reliability of each ofthe first arousal level and the second arousal level is the thresholdvalue or higher and the first arousal level is different from the secondarousal level, the arousal level estimation unit 15 may set the firstarousal level as the third arousal level and thereby estimate the thirdarousal level.

Note that in a case where the first arousal level is different from thesecond arousal level in the manual driving mode, the arousal levelestimation unit 15 performs a learning process by using the teacher datathat are generated based on the first arousal level.

In this embodiment, as illustrated in FIG. 1, the arousal levelestimation unit 15 includes an arousal level verification unit 151 andthe teacher data generation unit 152. The arousal level estimation unit15 estimates the third arousal level as final arousal level of thedriver based on the first arousal level that is obtained from the firstarousal level recognition unit 12 or the second arousal level that isobtained from the second arousal level recognition unit 17.

<<Arousal level Verification Unit 151>>

The arousal level verification unit 151 verifies the validity of thefirst arousal level that is obtained from the first arousal levelrecognition unit 12 or the validity of the second arousal level that isobtained from the second arousal level recognition unit 17. Here, as fora verification method of the validity of the first arousal level, in acase where the reliability of the arousal level included in theidentification results obtained from the first arousal levelidentification unit 122 is a predetermined threshold value or higher,the first arousal level may be assessed as valid. Further, as for averification method of the validity of the second arousal level, thesame method as the above-described first arousal level may be used.

The arousal level verification unit 151 outputs either one of the firstarousal level and the second arousal level as the final arousal level ofthe driver, that is, the third arousal level based on the verificationresults and the driving mode selected by the driving mode selection unit6. For example, in a case where the driving mode is the manual drivingmode, the arousal level verification unit 151 may output the firstarousal level as the third arousal level after confirmation of thevalidity of the first arousal level. Further, for example, in a casewhere the driving mode is the automated driving mode, the arousal levelverification unit 151 may output the second arousal level as the thirdarousal level after confirmation of the validity of the second arousallevel.

Further, in a case where the driving mode is the manual driving mode,the arousal level verification unit 151 performs a comparison aboutwhether the value of the first arousal level is different from the valueof the second arousal level. In a case where those are different, alearning trigger signal is output to the teacher data generation unit152.

<<Teacher Data Generation Unit 152>>

The teacher data generation unit 152 generates the teacher data from thefirst arousal level in the manual driving mode, outputs the teacher datato the second arousal level recognition unit 17, thereby causes thesecond arousal level identification unit 172 to perform the learningprocess. More specifically, in a case where the teacher data generationunit 152 receives the learning trigger signal from the arousal levelverification unit 151, the teacher data generation unit 152 generatesthe teacher data by using the first arousal level and outputs theteacher data to the second arousal level identification unit 172. Notethat the teacher data generation unit 152 generates the teacher datasuch that the teacher data conform to the arousal level identificationmodel that is used by the second arousal level identification unit 172.

Consequently, the teacher data generation unit 152 may cause the secondarousal level identification unit 172 to perform the learning process byusing the teacher data that are generated based on the first arousallevel. Note that, as described above, the learning process is performedby the second arousal level recognition unit 17 based on the teacherdata in a case where the second arousal level recognition unit 17acquires (receives) the teacher data from the teacher data generationunit 152.

[Action of Arousal level Estimation Device 10]

Next, a description will be made about an action of the arousal levelestimation device 10 that is configured as described earlier.

FIG. 5 is a flowchart that illustrates an outline of a process that isperformed by the arousal level estimation device 10 in this embodiment.

As illustrated in FIG. 5, the arousal level estimation device 10 firstdetects the driving information (S1). Note that the details aredescribed above and will thus not be described here. Next, the arousallevel estimation device 10 recognizes the first arousal level of thedriver from the driving information that is detected in S1 (S3). Next,the arousal level estimation device 10 detects the physiologicalinformation of the driver (S5). Next, the arousal level estimationdevice 10 recognizes the second arousal level of the driver from thephysiological information that is detected in S5 (S7). Finally, thearousal level estimation device 10 estimates the third arousal level ofthe driver in driving of the vehicle from at least one of the firstarousal level that is recognized in S3 and the second arousal level thatis recognized in S7 (S9). In S9, as described above, the arousal levelestimation device 10 estimates the third arousal level from the firstarousal level and the second arousal level in the manual driving modeand estimates the third arousal level from the second arousal level inthe automated driving mode.

Note that a process of S5 may be performed after a process of S1 or aprocess of S3 but may be performed simultaneously with the process of S1or the process of S3 or may be performed in parallel with the process ofS1. Further, a process of S7 may be performed in parallel with theprocess of S3 in a case where the process of S7 is performed after theprocess of S5.

[Action of Driving Support Device 1]

FIGS. 6A and 6B are flowcharts that illustrate an action that isperformed by the driving support device 1 in this embodiment. Note that,as described above, the driving support device 1 is capable of switchingthe automated driving mode in which the travel control of the vehicle isautomatically performed and the manual driving mode by the driver, inthe driving mode selection unit 6. Further, regardless of the drivingmode, the driving support device 1 regularly monitors the informationrelated to the surrounding environment around the traveling vehicle suchas the vehicular gap from the preceding vehicle by the surroundingenvironment recognition unit 3 or the subject vehicle position detectionunit 4.

Referring to FIGS. 6A and 6B, a description will first be made about anaction that is performed by the driving support device 1 in the manualdriving mode while the action of the arousal level estimation device 10is focused.

It is assumed that in S101, the arousal level estimation device 10assesses whether or not the driving mode is the manual driving mode andassesses the driving mode as the manual driving mode (Y in S101). Morespecifically, in a case where the manual driving mode is selected by thedriving mode selection unit 6, the arousal level estimation unit 15assesses the driving mode as the manual driving mode. In this case, thedriver performs driving of the vehicle, that is, the travel control ofthe vehicle by using the driving operation unit 2.

Next, the arousal level estimation device 10 detects the drivinginformation and the physiological information of the driver (S102).Specifically, while the driver is performing driving in the manualdriving mode, the vehicle behavior detection unit 11 detects the drivinginformation of the driving operation by the driver, and thephysiological information detection unit 16 detects the physiologicalinformation of the driver.

Next, the arousal level estimation device 10 extracts the drivinginformation features from the surrounding environment and the drivinginformation and thereby recognizes the first arousal level (S103).Specifically, the first arousal level recognition unit 12 recognizes thefirst arousal level of the driver by using the surrounding environmentof the subject vehicle that is obtained from the surrounding environmentrecognition unit 3 and the driving information that is obtained from thevehicle behavior detection unit 11. More specifically, as describedabove, the driving information feature extraction unit 121 extracts thedriving information features based on the surrounding environment of thesubject vehicle and the driving information, and the first arousal levelidentification unit 122 identifies the first arousal level by using thedriving information features that are extracted by the drivinginformation feature extraction unit 121.

Next, the arousal level estimation device 10 extracts the physiologicalinformation features from the physiological information and therebyrecognizes the second arousal level (S104). Specifically, the secondarousal level recognition unit 17 recognizes the second arousal level byusing the physiological information that is obtained from thephysiological information detection unit 16. More specifically, asdescribed above, the physiological feature extraction unit 171 extractsthe physiological information features based on the physiologicalinformation, and the second arousal level identification unit 172identifies the second arousal level by using the physiologicalinformation features that are extracted by the physiological featureextraction unit 171.

Next, the arousal level estimation unit 15 estimates the third arousallevel as the final arousal level of the driver based on the firstarousal level and the second arousal level (S105 and S106).Specifically, first, the arousal level verification unit 151 of thearousal level estimation unit 15 verifies the validity of the firstarousal level that is recognized in S103 (S105). Note that, as for theverification method of the validity of the first arousal level, in acase where the reliability of the arousal level included in theidentification results obtained from the first arousal levelidentification unit 122 is a predetermined threshold value or higher,the first arousal level may be assessed as valid. In S105, in a casewhere the first arousal level is assessed as valid (Y in S105), thefirst arousal level is output as the estimation result, that is, thethird arousal level (S106).

Next, the arousal level verification unit 151 verifies the validity ofthe second arousal level that is recognized in S104 (S107). Note that,as for the verification method of the validity of the second arousallevel, the same method as the above-described first arousal level may beused.

In S107, in a case where the second arousal level is assessed as valid(Y in S107), the comparison is performed between the first arousal leveland the second arousal level, and an assessment whether or not those aredifferent is performed (S108).

In S108, in a case where the first arousal level is different from thesecond arousal level (Y in S108), the arousal level verification unit151 outputs the result to the teacher data generation unit 152. In thisembodiment, in a case where the first arousal level is different fromthe second arousal level as a result of the comparison, the arousallevel verification unit 151 outputs the learning trigger signal to theteacher data generation unit 152.

Next, the teacher data generation unit 152 generates the teacher datafrom the first arousal level (S109) and outputs the generated teacherdata to the second arousal level recognition unit 17. In thisembodiment, in a case where the teacher data generation unit 152receives the learning trigger signal from the arousal level verificationunit 151, the teacher data generation unit 152 generates the teacherdata by using the first arousal level and outputs the teacher data tothe second arousal level identification unit 172. Note that the teacherdata generation unit 152 generates the teacher data such that theteacher data conform to the arousal level identification model that isused by the second arousal level identification unit 172.

Next, the second arousal level recognition unit 17 performs the learningprocess based on the teacher data that are obtained from the teacherdata generation unit 152 (S110). Then, the action returns to a processof S101, and the processes are repeated. In this embodiment, in a casewhere the second arousal level identification unit 172 acquires theteacher data from the teacher data generation unit 152, the secondarousal level identification unit 172 performs learning for the arousallevel identification model that is used for identifying the secondarousal level by using the teacher data. The action thereafter returnsto the process of S101.

Note that in a case where the second arousal level is not valid in S107(N in S107) or in a case where the first arousal level matches thesecond arousal level in S108 (N in S108), the learning process by thesecond arousal level recognition unit 17 is not performed, the actionreturns to the process of S101, and the processes are repeated.

Further, in S105, in a case where the first arousal level recognized inS103 is not assessed as valid (N in S105), the arousal levelverification unit 151 verifies the validity of the second arousal levelrecognized in S104 (S111). In S111, in a case where the second arousallevel recognized in S104 is assessed as valid (Y in S111), the arousallevel verification unit 151 outputs the second arousal level recognizedin S104 as the estimation result, that is, the third arousal level(S112). On the other hand, in S111, in a case where the second arousallevel recognized in S104 is not valid (N in S111), the arousal levelverification unit 151 outputs an error that indicates that the thirdarousal level may not be estimated (S113), and the action returns to theprocess of S101.

Referring to FIGS. 6A and 6B, a description will next be made about anaction that is performed by the driving support device 1 in theautomated driving mode while the action of the arousal level estimationdevice 10 is focused.

It is assumed that in S101, the arousal level estimation device 10assesses whether or not the driving mode is the manual driving mode andassesses the driving mode as not the manual driving mode (N in S101).More specifically, in a case where the automated driving mode isselected by the driving mode selection unit 6, the arousal levelestimation unit 15 assesses the driving mode as the automated drivingmode. In this case, the driver does not have to perform the drivingoperation, and the vehicle control unit 7 automatically performsautomated driving, that is, the travel control of the vehicle based onthe surrounding environment.

Next, the arousal level estimation device 10 detects the physiologicalinformation of the driver (S121). Specifically, in the automated drivingmode, the physiological information detection unit 16 detects thephysiological information of the driver. Here, in the automated drivingmode, because the driver himself/herself does not perform the drivingoperation, the vehicle behavior detection unit 11 may not obtaineffective driving information based on the driving operation by thedriver such as the steering angle in steering. On the other hand, thephysiological information detection unit 16 may obtain effectivephysiological information of the driver even in the automated driving.

Next, the arousal level estimation device 10 extracts the physiologicalinformation features from the physiological information and therebyrecognizes the second arousal level (S122). Specifically, the secondarousal level recognition unit 17 recognizes the second arousal level byusing the physiological information that is obtained from thephysiological information detection unit 16. More specifically, asdescribed above, the physiological feature extraction unit 171 extractsthe physiological information features based on the physiologicalinformation, and the second arousal level identification unit 172identifies the second arousal level by using the physiologicalinformation features that are extracted by the physiological featureextraction unit 171. In such a manner, the arousal level estimationdevice 10 recognizes the arousal level of the driver from thephysiological information of the driver not by using the first arousallevel recognition unit 12 but by using the second arousal levelrecognition unit 17.

Next, the arousal level estimation unit 15 estimates the third arousallevel as the final arousal level of the driver based on the secondarousal level (S123 and S124). Specifically, first, the arousal levelverification unit 151 of the arousal level estimation unit 15 verifiesthe validity of the second arousal level that is recognized in S122(S123). Note that, as for the verification method of the validity of thesecond arousal level, the same method as the above-described firstarousal level may be used. In S123, in a case where the second arousallevel is assessed as valid (Y in S123), the second arousal level isoutput as the estimation result, that is, the third arousal level(S124). Note that in a case where the second arousal level is not validin S123 (N in S123), the arousal level verification unit 151 outputs theerror that indicates that the third arousal level may not be estimated(S125), and the action returns to the process of S101.

[Effects]

As described earlier, this embodiment may realize an arousal levelestimation device that may highly precisely estimate the arousal levelof the driver of the vehicle that has the automated driving mode and themanual driving mode.

More specifically, the arousal level estimation device 10 in thisembodiment estimates the final estimation result, that is, the thirdarousal level by using the first arousal level that is recognized byusing the driving information of the driver and the second arousal levelthat is recognized by using the physiological information of the driverin the manual driving mode. Meanwhile, the arousal level estimationdevice 10 sets the second arousal level, which is recognized by usingthe physiological information of the driver, as the final estimationresult, that is, the third arousal level in the automated driving mode.That is, in the arousal level estimation device 10 in this embodiment,in the manual driving mode and the automated driving mode, the thirdarousal level of the driver is estimated by using the first arousallevel recognized by the first arousal level recognition unit 12 and thesecond arousal level recognized by the second arousal level recognitionunit 17 while switching is appropriately made between the first arousallevel and the second arousal level.

In addition, in the manual driving mode, the arousal level estimationdevice 10 in this embodiment causes the second arousal level recognitionunit 17 to perform the learning process by using recognition results ofthe first arousal level recognition unit 12 as the teacher data.Consequently, variations in the precision of recognition of the secondarousal level due to individual differences in the physiologicalinformation may be absorbed. Consequently, in the automated drivingmode, the arousal level estimation device 10 in this embodiment mayprecisely estimate the arousal level of the driver based on thephysiological information of the driver. That is, it is possible toprecisely estimate the arousal level of the driver even in the automateddriving at level 2 or higher in which the driving information such asthe driving operation by the driver or vehicle behavior may not be used.

As described in the foregoing, because the arousal level estimationdevice 10 in this embodiment may precisely estimate the arousal level ofthe driver based on the physiological information of the driver, thearousal level of the driver may precisely be detected even in anautomated driving system that includes the automated driving mode atlevel 2 or higher.

MODIFICATION EXAMPLE

In the above embodiment, the method is described in which the teacherdata generated by the teacher data generation unit 152 are used and thearousal level identification model itself that is used by the secondarousal level identification unit 172 is thereby learned. However,embodiments are not limited to this. The learning process may beperformed by using the first arousal level as the teacher data and byselecting the physiological information features that are effective foridentification of the second arousal level from plural physiologicalinformation features obtained from the physiological feature extractionunit 171. This case will be described as a modification example in thefollowing.

FIG. 7 is a block diagram that illustrates one example of aconfiguration of the driving support device in the modification example.Note that the same wording is used for similar contents to FIG. 1, and adetailed description will not be made. In the driving support device 1in the modification example illustrated in FIG. 7, the configuration ofa second arousal level recognition unit 17A of an arousal levelestimation device 10A is different from the driving support device 1 inthe above embodiment. Specifically, in the arousal level estimationdevice 10A in the modification example, a physiological informationfeature storage unit 173 and a physiological feature selection unit 174are added to the second arousal level recognition unit 17 in the aboveembodiment, and the configurations of a physiological feature extractionunit 171A and a second arousal level identification unit 172A aredifferent.

<Physiological Feature Extraction Unit 171A>

The physiological feature extraction unit 171A extracts thephysiological information feature from each of plural pieces ofphysiological information that are detected by the physiologicalinformation detection unit 16. The physiological feature extraction unit171A outputs the extracted physiological information features to thephysiological feature selection unit 174. Note that in the automateddriving mode, the physiological feature extraction unit 171A may outputthe extracted physiological information features to the second arousallevel identification unit 172A.

<Physiological Information Feature Storage Unit 173>

The physiological information feature storage unit 173 stores thephysiological information features that are obtained from thephysiological feature extraction unit 171A. The physiologicalinformation feature storage unit 173 may store the teacher data that areobtained from the teacher data generation unit 152.

<Physiological Feature Selection Unit 174>

In the manual driving mode, the physiological feature selection unit 174selects the physiological information features, which are highlycorrelated with the teacher data generated from the first arousal levelby the teacher data generation unit 152, among the plural physiologicalinformation features extracted by the physiological feature extractionunit 171A.

More specifically, in the manual driving mode, the physiological featureselection unit 174 stores teacher data D(t) (t represents the time) thatare obtained from the teacher data generation unit 152 in a prescribedstorage unit for predetermined times (m times) as teacher data seriesD(1), D(2), . . . , D(m). In this modification example, a descriptionwill be made in the following on an assumption that the teacher dataD(t) are stored in the physiological information feature storage unit173. Further, in the manual driving mode, the physiological featureselection unit 174 stores n kinds of physiological information featuresBi(t) (i=1 to n) that are obtained from the physiological featureextraction unit 171A in the physiological information feature storageunit 173 for predetermined times (m times) as physiological informationfeature data series Bi(1), Bi(2), . . . , Bi(m).

In the manual driving mode, for example, in a case where a predeterminednumber of data series are stored in the physiological informationfeature storage unit 173, the physiological feature selection unit 174calculates the correlation coefficient between the teacher data seriesD(t) (t=1, 2, . . . , m) stored in the physiological information featurestorage unit 173 and the n kinds of physiological information featuredata series Bi(t) (t=1, 2, . . . , m) stored in the physiologicalinformation feature storage unit 173. Note that the correlationcoefficient may be calculated by a method in related art such ascalculation by dividing the covariance between D(t) and Bi(t) by theproduct value of the standard deviation of D(t) and the standarddeviation of Bi(t), for example.

In the manual driving mode, the physiological feature selection unit 174compares the calculated correlation coefficient with a predeterminedthreshold value, selects the physiological information features whosecorrelation coefficient is the threshold value or higher, and outputsthe physiological information features to the second arousal levelidentification unit 172A. Note that in the automated driving mode, thephysiological feature selection unit 174 may output the physiologicalinformation features that are selected in the manual driving mode to thesecond arousal level identification unit 172A.

<Second Arousal level Identification Unit 172A>

In the automated driving mode and the manual driving mode, the secondarousal level identification unit 172A identifies the second arousallevel by using the physiological information features that are selectedby the physiological feature selection unit 174.

In the manual driving mode, the second arousal level identification unit172A performs the learning process based on the teacher data generatedby the teacher data generation unit 152. In this modification example,in a case where the second arousal level recognition unit 17A receivesthe teacher data from the teacher data generation unit 152, the secondarousal level identification unit 172A performs the learning process byusing the physiological information features that are selected by thephysiological feature selection unit 174 as the teacher data. Note thatin a case where the kinds of physiological information features, whichare the physiological information features selected by the physiologicalfeature selection unit 174 and are used for identification of the secondarousal level, are changed, the second arousal level identification unit172A has to generate (learn) the arousal level identification model byusing only the selected physiological information features.

[Effects]

As described earlier, this modification example may realize an arousallevel estimation device that may highly precisely estimate the arousallevel of the driver of the vehicle that has the automated driving modeand the manual driving mode.

More specifically, in the arousal level estimation device 10A in thismodification example, in the manual driving mode, the physiologicalfeature selection unit 174 selects the physiological informationfeatures, which are highly correlated with the first arousal level asthe teacher data, among the plural kinds of physiological informationfeatures that are obtained from the physiological feature extractionunit 171A. Further, the second arousal level may be identified by usingthe arousal level identification model that performs the learningprocess with the selected physiological information features as theteacher data.

In such a manner, the arousal level estimation device 10A in thismodification example may generate (learn) the arousal levelidentification model while selecting physiological information featuresthat are effective for identification of the arousal level and may thusabsorb variations due to individual differences in the physiologicalinformation.

In addition, in this modification example, the second arousal levelidentification unit 172A may reduce the processing amount compared to acase where the second arousal level is identified by using all thephysiological information features that are obtained from thephysiological feature extraction unit 171A and may thus increase theprocessing speed of the second arousal level identification unit 172A.

Other Embodiments

The present disclosure is not limited to the above embodiment. Forexample, other embodiments that are realized by arbitrarily combiningthe configuration elements described herein or omitting severalconfiguration elements may be provided as embodiments of the presentdisclosure. Further, the present disclosure includes modificationexamples that are obtained by applying various modifications conceivedby persons having ordinary skill in the art within the scope which doesnot depart from the gist of the present disclosure in relation to theabove embodiment, that is, the meanings indicated by the wordingdescribed in the claims.

For example, in the above-described driving support device 1, eachconfiguration element in each of the units such as the surroundingenvironment recognition unit 3, the subject vehicle position detectionunit 4, the target travel state decision unit 5, the driving modeselection unit 6, the vehicle control unit 7, the notification unit 8,and the vehicle behavior detection unit 11, the first arousal levelrecognition unit 12, the physiological information detection unit 16,the second arousal level recognition unit 17, and the arousal levelestimation unit 15, which are included in the arousal level estimationdevice 10, may be configured with dedicated hardware. Further, theconfiguration element may be realized by executing a software programthat is suitable for each of the configuration elements. Further, aprogram execution unit such as a CPU or a processor reads out andexecutes a software program that is recorded in a recording medium suchas a hard disk or a semiconductor memory, and each of the configurationelements may thereby be realized.

Further, the present disclosure also includes cases described in thefollowing.

(1) The above devices are computer systems that are specificallyconfigured with a microprocessor, a ROM, a RAM, a hard disk unit, adisplay unit, a keyboard, a mouse, and so forth. The RAM or the harddisk unit stores a computer program. The microprocessor acts inaccordance with the computer program, and the devices thereby achievetheir functions. Here, the computer program is configured by combiningplural instruction codes that indicate commands for a computer in orderto achieve a prescribed function.

(2) In addition, a portion of or all configuration elements thatconfigure the above devices may be configured with one system largescale integration (LSI). A system LSI is a super multi-function LSI thatis manufactured by integrating plural configuration units on one chipand is specifically a computer system configured to include amicroprocessor, a ROM, a RAM, and so forth. The RAM stores a computerprogram. The microprocessor acts in accordance with the computerprogram, and the system LSI thereby achieves its function.

(3) A portion of or all configuration elements that configure the abovedevices may be configured with IC cards or individual modules that areremovable from the devices. The IC card or the module is a computersystem that is configured with a microprocessor, a ROM, a RAM, and soforth. The IC card or the module may include the above supermulti-function LSI. The microprocessor acts in accordance with acomputer program, and the IC card or the module thereby achieves itsfunction. The IC card or the module may be tamper-resistant.

(4) Further, one embodiment of the present disclosure may be the methodsthat are described above. Further, one embodiment of the presentdisclosure may be a computer program that realizes those methods by acomputer or digital signals that are configured with the computerprogram.

(5) Further, one embodiment of the present disclosure may be thecomputer program or the digital signals that are recorded incomputer-readable recording media such as a flexible disk, a hard disk,a CD-ROM, an MO, a DVD, a DVD-ROM, a DVD-RAM, a Blu-ray® disc (BD), or asemiconductor memory, for example. Further, one embodiment of thepresent disclosure may be the digital signals that are recorded in thoserecording media.

(6) Further, one embodiment of the present disclosure may be thecomputer program or the digital signals that are transmitted via anelectric communication line, a wireless or wired communication line, anetwork represented by the Internet, data broadcasting, and so forth.

(7) Further, one embodiment of the present disclosure may be a computersystem that includes a microprocessor and a memory, in which the memorystores the computer program and the microprocessor acts in accordancewith the computer program.

(8) Further, one embodiment of the present disclosure may be practicedby another independent computer system by transferring the recordingmedia that record the program or the digital signals or by transferringthe program or the digital signals via the network or the like.

(9) Further, all the numerals used in the above are exemplified forspecifically describing the present disclosure, and the presentdisclosure is not restricted by the exemplified numerals.

(10) Further, the divisions of the function blocks in the block diagramsare examples. Plural function blocks may be realized as one functionblock, one function block may be divided into plural function blocks, ora portion of functions may be moved to another function block. Further,the functions of plural function blocks that have similar functions maybe processed by a single piece of hardware or software in parallel or ina time-division manner.

(11) Further, the order of execution of plural steps included in theabove arousal level estimation method is for exemplification forspecifically describing the present disclosure and may be another orderthan the above. Further, a portion of the above steps may simultaneously(in parallel) be executed with the other steps.

<<Supplement>>

Each of the various arousal level estimation devices that are describedin the above embodiments and modification example is one example of“system for assessing arousal level of a driver of a vehicle” accordingto the present disclosure. Each of the arousal level estimation methodsthat are described in the above embodiments and modification example isone example of “method for assessing arousal level of a driver of avehicle” according to the present disclosure.

The arousal level assessment device according to one aspect of thepresent disclosure may not include sensors such as the vehicle behaviordetection unit and the physiological information detection unit and mayreceive the driving information and the physiological information fromsensors that are provided on the outside of the arousal level assessmentdevice, for example. In this case, the arousal level assessment deviceincludes a memory that records programs and a processor, for example,and the processor executes assessment of the arousal level of the driverin accordance with the programs read out from the memory. In addition,the arousal level assessment device according to one aspect of thepresent disclosure may receive driving features extracted from thedriving information and/or physiological features extracted from thephysiological information from outside sensors, for example.

“First mode” according to one aspect of the present disclosure may be amode in which control in both of the front-rear direction and theleft-right direction of the vehicle is performed by the driver, forexample. “Second mode” may be a mode in which control of at least one ofthe front-rear direction and the left-right direction of the vehicle isperformed automatically, for example.

An arousal level assessment system and an arousal level assessmentmethod according to one aspect of the present disclosure may beinstalled in a driving support system or the like that estimates arousallevel of a driver in driving and alerts the driver in a case where thearousal level becomes lower than a prescribed value.

What is claimed is:
 1. An apparatus for assessing arousal level of adriver of a vehicle, the apparatus comprising: a processor; and a memorystoring a computer program, which when executed by the processor, causesthe processor to perform operations including (A) acquiring driving modeinformation that indicates a driving mode of the vehicle is either afirst mode, in which travel control of the vehicle is performed by thedriver, or a second mode, in which at least a part of the travel controlof the vehicle is automatically performed, (B) determining whether thedriving mode of the vehicle is either the first mode or the second modebased on the driving mode information, (C) executing the following (c1)to (c3) in a case where the driving mode is determined as being thefirst mode, (c1) acquiring driving information that indicates a drivingoperation by the driver and/or indicates a driving state of the vehiclevia at least one first sensor that is mounted on the vehicle, (c2)acquiring physiological information of the driver via at least onesecond sensor that is mounted on the vehicle or is attached to thedriver, and (c3) determining the arousal level based on both of thedriving information and the physiological information, and (D) executingthe following (d1) and (d2) in a case where the driving mode isdetermined as being the second mode, (d1) acquiring the physiologicalinformation via the at least one second sensor, and (d2) determining thearousal level based on the physiological information without referringto the driving information.
 2. The apparatus according to claim 1,wherein the (c3) includes (c31) calculating first arousal level of thedriver using the driving information, (c32) calculating second arousallevel of the driver using the physiological information, and (c33)determining the arousal level by referring to the first arousal leveland the second arousal level.
 3. The apparatus according to claim 1,wherein the (c3) includes (c31) calculating first arousal level of thedriver using the driving information, (c32) calculating second arousallevel of the driver using the physiological information, and (c33)determining, as the arousal level, either the first arousal level or thesecond arousal level.
 4. The apparatus according to claim 3, wherein the(c33) includes (c331) assessing whether or not reliabilities of thefirst arousal level and the second arousal level each are equal to orhigher than a threshold value, and (c332) selecting one of the firstarousal level and the second arousal level of which reliability is equalto or higher than the threshold value, as the arousal level.
 5. Theapparatus according to claim 4, wherein, in the (c332), the firstarousal level is selected as the arousal level, in a case where each ofthe reliabilities of the first arousal level and the second arousallevel is equal to or higher than the threshold value.
 6. The apparatusaccording to claim 2, wherein the (d2) includes inputting the acquiredphysiological information into a model to calculate the arousal level,and the operations further includes (E) updating the model using thefirst arousal level and the physiological information, both of whichreflect a state of the driver at a time.
 7. The apparatus according toclaim 1, wherein the processor, in the (c3), generates or updates amodel that indicates a relationship between the physiologicalinformation and the arousal level.
 8. The apparatus according to claim1, wherein the at least one second sensor includes a heart rate sensorthat detects a heart rate of the driver.
 9. The apparatus according toclaim 1, wherein the at least one second sensor includes a camera thatdetects a facial expression of the driver.
 10. The apparatus accordingto claim 1, wherein the at least one second sensor includes a bodymovement sensor that detects body movement of the driver.
 11. Theapparatus according to claim wherein the at least one first sensorincludes a steering angle sensor that detects a steering angle of asteering wheel of the vehicle.
 12. The apparatus according to claimwherein the at least one first sensor includes a position sensor thatdetects positions of an accelerator pedal and/or a brake pedal of thevehicle.
 13. The apparatus according to claim 1, wherein the at leastone first sensor includes an acceleration sensor that detects anacceleration of the vehicle.
 14. The apparatus according to claim 1,wherein, in the (c1), the driving information includes information aboutthe at least a part of the travel control performed by the driver.
 15. Avehicle, comprising: a controller that performs travel control of thevehicle in either the first mode or the second mode; and the apparatusaccording to claim 1.